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Identifying schools that are beating the odds—schools that are exceeding academic 
performance expectations, given the composition of their student body—can lead to 
promising practices that other schools serving similar populations can implement. 
Regional Educational Laboratory Northeast & Islands used data on high-poverty high 
schools in Puerto Rico to show how two methods of identifying beating-the-odds high 
schools led to the identification of two sets of schools with relatively little overlap. These 
results suggest that it may be useful for state and local education agencies to consider 
both methods for identifying beating-the-odds schools. 


Why this study? 


During the past decade states and school districts have focused increasingly on turning around per 
sistently low-performing schools (Duke, Tucker, Salmonowicz, Levy, & Saunders, 2007; Herman et al., 
2008). As of 2012 there were approximately 5,000 chronically low-performing schools nationwide that 
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were not providing students with an education that prepares them for lifelong success (U.S. Department of 
Education, 2012). Students in these schools have lower scores on assessments and graduate at lower rates 
than do students at other schools. At the secondary level such schools are sometimes called “dropout fac- 
tories” (Balfanz & Legters, 2004; 2006). Federal policy initiatives, such as School Improvement Grants and 
Race to the Top, have provided funding to help improve academic outcomes, in particular for the lowest 
performing schools (Council of the Great City Schools, 2015). 


State and local education agencies are under pressure to identify and improve their lowest performing 
schools by developing and implementing supports, initiatives, and interventions to raise academic perfor 
mance (Le Floch et al., 2014). Many of these efforts seem to operate under the assumption that successful 
supports, initiatives, and interventions in high-performing schools are transferable to demographically sim- 
ilar low-performing schools (Murphy & Meyers, 2008). 


The challenge of turning schools around is also prevalent in Puerto Rico, where according to the 2013 
National Assessment of Educational Progress results, only 5 percent of students in grade 8 performed at or 
above the basic level in math, and less than 1 percent scored at or above the proficient level (U.S. Depart- 
ment of Education, n.d.). Moreover, only 62 percent of students who entered public high schools in Puerto 
Rico graduated on time in 2011/12, compared with a national average of 82 percent (Stetser & Stillwell, 
2014). Turning around low-performing schools is thus increasingly a priority for educators and policymakers 
in Puerto Rico. 


In response to a request from the Puerto Rico Research Alliance for Dropout Prevention, Regional Edu- 
cational Laboratory Northeast & Islands compared two methods of identifying beating-the-odds schools 
—schools that exceed academic performance expectations, given the characteristics of the students 
enrolled—in Puerto Rico. This study may be useful to Puerto Rico stakeholders and state- and district-level 
education leaders elsewhere as they consider how to develop and establish statistical methods that reliably 
identify beating-the-odds schools. The identification of beating-the-odds schools in this report can provide 
initial information to the Puerto Rico Department of Education as it plans how to examine processes and 
practices of beating-the-odds schools. 


What the study examined 


The study used school-level data to identify beating-the-odds public high schools in Puerto Rico from 
among a sample of 159 high-poverty high schools (schools with at least 40 percent of students from low- 
income households). The study focused on the performance of the cohort of students who were expect 
ed to graduate in the 2012/13 school year. Beating-the-odds schools were identified using two methods: a 
status method and an exceeding achievement expectations method. Under the status method schools were 
ranked on their observed performance on two outcome measures: 2012/13 graduation rates and 2011/12 
grade 11 combined proficiency rates on reading and math assessments. Under the exceeding achievement 
expectations method schools were ranked based on the extent to which their observed performance on 
these outcome measures exceeded or fell short of their expected levels of achievement on the same two 
outcome measures. 


The study also compared the schools identified using each method in order to determine agreement rates 
—defined as the “ratio of the number of schools that appear on both sets of the lists to the average number 
of schools across the two lists” (Abe et al., 205, p. B-7; see box 1 for a discussion of data and methods). The 
report compares the performance of beating-the-odds schools identified using each method and all 159 
high-poverty high schools in Puerto Rico. 


Box 1. Data and methods for identifying beating-the-odds schools 


Data and sample. Publicly available school-level demographic data were retrieved from the U.S. Department 
of Education (2010, 2013, 2014). Graduation rates, poverty rates, and assessment results were provided 
by the Puerto Rico Department of Education. The analytic sample that was used to identify beating-the-odds 
schools included all regular public high schools in Puerto Rico operating during 2011/12 and 2012/13 with at 
least 40 percent of students from low-income households (vocational schools and alternative schools were not 
included). The sample comprised 159 schools. 


Methodology for identifying and comparing beating-the-odds schools. The analysis focused on the performance 
of the 2012/13 graduation cohort on two outcome measures: the 2012/13 cohort graduation rate and the 
2011/12 grade 11 proficiency rate for reading (in Spanish, the language of instruction in Puerto Rico) and math 
combined, weighted by the number of students tested in each subject. 

Two methods were used to identify beating-the-odds schools: ranking by status and ranking by exceeding 
achievement expectations. 

e¢ The status method ranked schools on the basis of their observed performance on each of the outcome 
measures. Schools that ranked among the top 25 percent on both outcome measures were identified as 
beating-the-odds schools. 

e The exceeding achievement expectations method estimated schools’ expected performance on each 
outcome measure using ordinary least squares regressions that controlled for grade 8 achievement levels, 
school poverty rate, and other student and school characteristics that might be related to school perfor- 
mance. As a proxy for prior achievement, the study used the weighted average school-level proficiency rate 
in grade 8 reading and math of feeder schools* in 2008/09—the year the 2012/13 graduating cohort was 
in grade 8—as an independent variable. The high schools were ranked based on the differences between 
their observed performance and expected performance (performance net of expectations). Schools that 
ranked in the top 25 percent on both outcomes were identified as beating-the-odds schools. 

The report includes ranks for each beating-the-odds school on each outcome measure (2012/13 graduation 

rate and 2011/12 combined reading and math proficiency rate) for both methods. Correlation coefficients 

were calculated between each outcome across methods. Differences and similarities between the two lists of 

beating-the-odds schools were measured by computing an agreement rate (the ratio of the number of schools 

that appear on both lists to the average number of schools across the two lists). Finally, the report presents the 

performance of beating-the-odds schools as compared with that of all high-poverty high schools in Puerto Rico. 
Additional information about the data and methodology can be found in Meyers & Wan (2016). 

Note 


1. School feeder patterns designate the schools that students attend as they matriculate from one level to the next. The patterns 
are usually determined by the location of the students’ residence and its location within the school boundary. Because the actual 
feeder patterns from middle schools to high schools in Puerto Rico are not known, the study group identified possible feeder 
schools for each high school in the sample using information on region, district, school location, and distance between schools. 


What the study found 


Using the two methods—ranking by status and by exceeding achievement expectations—the study iden- 
tified two lists of beating-the-odds schools. The agreement rate between the two lists was 38 percent. This 
section further describes the findings. 


The rankings by the two methods—status and exceeding achievement expectations—resulted in different lists of 
beating-the-odds schools 


The status method identified 17 beating-the-odds schools, and the exceeding achievement expectations 
method identified 15 schools. Six schools were identified as beating the odds by both methods. 


The agreement rate between the two methods on each criterion for identifying beating-the-odds schools 
was 58 percent for the graduation rate and 68 percent for the proficiency rate (table 1). But when the study 
team identified the schools on the two lists that met both criteria for each method (the beating-the-odds 
schools), the agreement rate was just 38 percent. The relatively low agreement rate for schools identified 
using the two methods illustrates how the use of different methods—ranking by status versus ranking by 
exceeding achievement expectations—affects the school identification results. 


School performance varied by method and outcome measure 


Beating-the-odds schools identified using the status method had a higher average graduation rate and profi- 
ciency rate than beating-the-odds schools identified using the exceeding achievement expectations method. 
The average graduation rate in 2012/13 was 84 percent for beating-the-odds schools identified using the 
status method and 77 percent for beating-the-odds schools identified using the exceeding achievement 
expectations method. Both rates were higher than the average for all schools in the sample (65 percent; 
figure 1). The average grade 11 proficiency rate was 47 percent for beating-the-odds schools identified using 
the status method and 36 percent for beating-the-odds schools identified using the exceeding achievement 
expectations method. The average was 24 percent for all schools in the sample. 


Schools that were identified as beating the odds by the status method ranked lower when analyzed by 
the exceeding achievement expectations method. When the study compared the observed performance of 
schools identified using the exceeding achievement expectations method with their expected performance 
after accounting for students’ prior achievement and demographic characteristics, the average differenc- 
es (observed graduation rate or proficiency rate minus expected graduation rate or proficiency rate) were 
15 percentage points for graduation rate and 16 percentage points for grade 11 proficiency rate (figure 2). 
This discrepancy was expected because the exceeding achievement expectations method, by definition, 
identified schools that ranked at the top on performance-net-of-expectations differences. 


The average difference between observed and expected performance for beating-the-odds schools identified 
using the status method was 8.8 percentage points for graduation rate and 9.3 percentage points for profi- 
ciency rate (see figure 2). This suggests that on average the schools identified using the status method also 
performed better than expected based on their background characteristics. However, some of the schools 
(three for graduation rate and four for proficiency rate) performed worse than expected, with the difference 
between observed and expected performance in proficiency rate as large as 15.4 percentage points. 


Table 1. Agreement rate on outcome measures for Puerto Rico high schools identified as beating 
the odds, by identification method, 2011/12 and 2012/13 


Number of schools identified 


Exceeding Agreement 
achievement rate between 

Status expectations Both methods 
Outcome measure Criterion method method methods (percent) 
Graduation rate, 2012/13 Top 25 percent 40 40 23 58 
Grade 11 proficiency rate (reading and 
math combined), 2011/12 Top 25 percent 40 40 27 68 
Both measures (beating-the-odds schools) Met all criteria 17 15 6 38 


Note: The total sample size is 159 schools. The number of students in the cohort by school ranged from 7 to 357. The analyses 
used data on the 2012/13 graduation rate, 2011/12 grade 11 combined reading and math proficiency rate, 2008/09 grade 8 
combined reading and math proficiency rate, 2009/10 school poverty rate, and data on several other school characteristics for the 
2012/13 school year. 


Source: Authors’ analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the Puerto Rico Department 
of Education. 


Figure 1. Average observed performance of beating-the-odds schools was higher for schools 
identified using the status method, 2011/12 and 2012/13 
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Source: Authors’ analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the Puerto Rico Department 
of Education. 


Figure 2. Average differences between observed and expected performance of beating-the-odds 
high schools in Puerto Rico were higher for the exceeding achievement expectations method, 
2011/12 and 2012/13 
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Note: The difference between the observed and expected performance indicates how much better or worse a school actually per- 
formed than its characteristics predicted it would. The height of the bars indicates the mean difference for each group—that is, on 
average, how much each group of schools performed better than expected. By definition, the mean differences between observed 
and expected performance for all schools in the sample are zero under the exceeding achievement expectations method. 


Source: Authors’ analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the Puerto Rico Department 
of Education. 


The findings suggest a potential issue with using the status method: looking at status alone may identify 
schools that did not perform better than expected (and potentially performed worse than expected) based 
on their school and student characteristics. 


Implications for research and policy and limitations of the study 


This report highlights how identification of beating-the-odds schools can be affected by methodological 
choices. Seventeen schools were identified by the status method, 15 schools were identified by the exceed- 
ing achievement expectations method, and 6 schools were identified by both methods (38 percent agree- 
ment rate). The analyses suggest that identifying schools with high percentages of students in poverty and 
high performance (the status method) is a straightforward method of identifying beating-the-odds schools 
but that considering the exceeding achievement expectations method as well will ensure that schools that 
are doing better than other schools with similarly challenging circumstances are identified. 


The exceeding achievement expectations method adjusts for the potential influence of school background 
factors, including student prior achievement. By statistically controlling for these factors, this method may 
produce less biased estimates of school performance than does the status method. But some schools identi- 
fied using the exceeding achievement expectations method have relatively low overall graduation rates or 
proficiency rates, which should be no surprise because schools identified using this method are by design 
not as highly ranked. 


The status method identified beating-the-odds schools that ranked in the top 25 percent on observed 
performance among schools with at least 40 percent of students from low-income households. However, 
some schools identified using the status method ranked low when the exceeding achievement expectations 
method was used (and when school poverty level and other student and school characteristics were statis- 
tically controlled for). These findings suggest that looking at status alone may identify schools that did not 
actually perform better (and potentially performed worse) than expected based on their school and student 
characteristics. 


It may thus be useful to consider both methods for identifying beating-the-odds schools. The high schools 
identified by both methods had high levels of performance and appeared to be achieving higher graduation 
rates and proficiency rates than might be expected given their demographics. 


Still, the analyses using the two methods to identify beating-the-odds schools were limited by the data 
provided on the performance and characteristics of Puerto Rico high schools. The analyses used grad- 
uation rate data for only one student cohort and academic achievement data for only one year because 
they were the only cohort and year with complete data available. Although school performance tends to 
show stability over time (Dumay, Coe, & Anumendem, 2014), it may be affected by cohort- or year-specific 
factors. If at all possible, analyses such as these should be informed by using multiple years of performance 
data. The analysis for the exceeding achievement expectations method was limited by the data provided 
on the characteristics of Puerto Rico high schools. Additional school and community characteristics not 
examined in this study, such as parents’ education level or parent or community involvement, also might 
be important factors affecting school success. Having multiple years of performance data or having data for 
a more complete set of school characteristics may improve model specifications for the exceeding achieve- 
ment expectations method and produce unbiased estimates of school performance. 
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The National Center for Education Evaluation and Regional Assistance (NCEE) conducts unbiased 
large-scale evaluations of education programs and practices supported by federal funds; provides 
research-based technical assistance to educators and policymakers; and supports the synthesis and 
the widespread dissemination of the results of research and evaluation throughout the United States. 
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